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MULTILEVEL IMAGE GRID DATA STRUCTURE AND IMAGE SEARCH METHOD 
USING THE SAME 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention relates to an image grid data structure and an image 
search method using the same, and in particular to a multilevel image grid data 
structure having a structure of different hierarchical grid levels with respect to one 
color feature related to a spatial color property of a still image and an image search 
method for searching an image using a multilevel image grid data structure. 

2. Description of the Background Art 

In a conventional image search method, a feature such as a color, shape, 
texture, etc. is expressed in an image grid data structure of one level, and a similarity 
between different image data of the same structure is searched using an image grid 
data of one level for thereby searching the image. 

When searching an image in accordance with a conventional image search 
method, the importance of each feature is different in accordance with the 
characteristics of an image which will be searched. In addition, even with respect to 
only one feature, the importance is different for each ceil in the conventional image 
grid data structure. For example, in the image search method using a color histogram, 
which is formed in a n-dimensional structure, a weight value reflecting the importance 
of each element can be determined as a different value for each element forming the 
n-dimensional structure. 
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Namely, in the conventional image search method using an image data 
stucture of one level, the importance between features is expressed based on the 
corresponding grid. In this case, however, the importance for each element of a 
certain feature is not considered. In order to resolve this problem, another 

5 conventional image search method adopts a method for computing an average 
importance of the elements in a certain feature. 

However, in the above-described conventional image search method, the 
average importance for elements of a certain feature is not useful, i.e., a pre- 
determination of an average value for elements of a certain feature is not useful in 

10 image search since the importance of each element caries by a reference image of 
target image. 

In addition, since the conventional image grid data structure is formed of only 
one level, the destination contained in an image (or target image) is not accurately 
searched in the conventional image search method. 

15 

SUMMARY OF THE INVENTION 

Accordingly, it is an object of the present invention to provide a data structure in 
which each level is expressed by the cells of a hierarchical structure of different levels 
20 by expressing one feature based on a multilevel image grid, and expressing a region 
representative color of each cell and a reliability with respect to the region 
representative color. 

It is another object of the present invention to provide an image search method 
capable of matching between cells of the same level of two image grids, different 
25 levels of grids, and color regions to perform a color similarity retrieval with respect to 
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multilevel image grids corresponding to different images. 

To achieve the above objects, there is provided a multilevel image data 
structure according to the present invention in which a spatial color feature of one 
image is expressed in a hierarchical image grid structure having more than two 
5 different levels. 

To achieve the above objects, there is provided an image search method using 
a multilevel image data structure according to the present invention in which the color 
similarities of a spatial color feature of a reference image divided into different 
hierarchical image grid levels and a target image are matched, so that an image is 
%o searched in accordance with user's content-based query. 

Additional advantages, objects and features of the invention will become more 
!:f apparent from the description which follows. 

P BRIEF DESCRIPTION OF THE DRAWINGS 

yli5 

O The present invention will become more fully understood from the detailed 

description given hereinbelow and the accompanying drawings which are given by 
way of illustration only, and thus are not limitative of the present invention, and 
wherein: 

20 Figure 1 is a view illustrating an embodiment of a multilevel image grid data 

structure and a 3-level image grid data structure according to the present invention; 

Figure 2 is a view illustrating an image search method using a multilevel image 
grid data structure and the construction of a match between 3-level image grid data 
structures according to the present invention; 

25 Figure 3 is a view illustrating an embodiment of an image search method using 
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a multilevel image grid data structure and the construction of a match between the 
same levels in a 3-level image grid data structure according to the present invention; 

Figure 4 is a view illustrating an embodiment of an image search method using 
a multilevel image grid data structure and the construction of a match between 
5 different levels of a 3-level image grid data structure according to the present 
invention; and 

Figures 5A and 5B are views illustrating an embodiment of an image search 
method using a multilevel image grid data structure according to the present invention, 
of which Figure 5A is a view illustrating two same image grid data structures, and 
lio Figure 5B is a view illustrating a process of a match of two image grid data structures. 

jj DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

p The present invention relates to a multilevel image grid data structure and an 

fks image search method using the same. The method for generating a multilevel image 
Q grid data structure according to the present invention will be explained. 

In the case of square image, it is uniformly divided by height and width, and in 
the case of a non-square image, one side is uniformly divided in accordance with an 
aspect ratio of a width and height of an image, and the other side is uniformly divided 
20 by the unit of one side. Namely, a regular square structure having the same length of 
horizontal and vertical sides is divided by the same unit, and in the case of a 
rectangular structure having different lengths of horizontal and vertical sides, one 
side(for example, a lengthy side) is uniformly divided, and the other side{for example, 
a shorter side) is divided by the dividing unit of one side. 
25 Therefore similarly as above, in one image data structure, the spatial color 
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feature is divided into hierarclilcai grids of different levels for thereby expressing a 

structure of a multilevel image grid. 

At this time, each image grid is a hierarchical structure of different levels, and 

the resolution of each level is hierarchically divided. The cell of each grid is assigned 
5 with two values which are a regional representative color (RRC) and a reliability score 

(S) relating to an accuracy of the regional representative color. 

Figure 1 illustrates an embodiment of a multilevel image grid data structure and 

a 3-ievel image grid data structure according to the present invention. Namely, one 

image is expressed in an image grid level of a first level, second level, and third level. 
10 In the resolution of the 3-level image grid data structure, the first level image 

grid is the lowest, the second level image grid is an intermediate level, and the third 

level image grid is higher than the second level image grid in accordance with the 

divided levels. 

The first level image grid is divided into the image region including a M1xN1 
15 number of local cells in proportion to the aspect ratio of a vertical side M and a 
horizontal side N. Each cell is expressed as a region representative color(RRC) which 
represents each region, and a reliability score(S) which corresponds to the accuracy 
of the representative color value. 

In addition, the second level image grid and the third level image grid are 
20 divided into the image regions including a M2xN2 number and M3xN3 number of local 
cells in accordance with the dividing state, and each cell has a region representative 
color(RRC) and a reliability score(S). 

For example, when the maximum vertical length M of the first level image grid 
and the horizontal length N are 8(=8x8), the maximum vertical length M2 of the 
25 second level image grid and the horizontal length N2 are 16(=16x16), and the 
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maximum horizontal length M3 of the third level image grid and the vertical length N3 
are 32(=32x32) of the local cells. 

Here, a certain cell Cell(i j) of the third level image grid is expressed as a region 
representative color and a reliability score C^y, S^y. 

At this time, the number of divisions of each of the image levels of 1st level, 
second level and third level is determined based on an aspect ratio of the image for 
accurately expressing the position of the object included in the image. Namely, in the 
case of the lengthy side, the lengthy side is uniformly divided, and the short side is 
divided by the divided unit of the lengthy side. 

In another method for generating the grid of the image, to increase processing 
speed and to consider approximate positional information of the object included in the 
image, the vertical and horizontal lengths may set identically. 

The image search method using the multilevel image grid data structure will be 
explained. 

Different images divided into the multilevel image grids are expressed as a 
representative region color(RRC) which represents the region and a reliability score 
which expresses an accuracy of the representative color, and a pair of representative 
region color and reliability are matched to another one, and a cell similarity is 
computed in accordance with the content-based query of a user for thereby 
performing an image search. 

The color similarity between two images is computed using the multilevel 
image grid data structure by comparing the cells included in an image grid of each 
level and the region color(RRC) representing each cell. Namely, the color similarity 
between two cells is computed using the color similarities Color__Sim(RRC_C1, 
RRC_C2) which represent the similarity of a region representative color value 
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between the cell CI and Cell C2. 

The first weight (a) is multiplied by the color similarities Color_Sim(RRC_C1, 
RRC_C2), and a result of the multiplication of the color similarities 
Color_Sim(RRC_C1, RRC_C2) and the second weight (p) and the similarity I with 
respect to a reliability between two cells is summed by the result obtained by 
multiplying the color similarity and the first weight. The thusly summed value is divided 
by the first weight and second weight and then is normalized, so that the cell similarity 
CelLSim(C1 , C2) of two cells C1 , C2 is obtained. The above-described operation may 
be expressed as follows. 



Cell &>„(Cl,C2)=<«^^''-'>'<^°^'--^^<^^g-"-^^g-"' — (1) 



I Here, the similarity I of the reliability(S1, S2) between two cells is obtained 

i based on [=1-|S1-S2|. 

is Therefore, the cell similarities between two different multilevel image grid are 

matched with respect to the portions between the same levels of the multilevel image 
and the different levels, and a feature between the images is compared. 

Figure 2 illustrates an embodiment of the image search using a multilevel 
image grid data structure according to the present invention and a similarity-based 

20 search between the grids of two images \^ and I2 having a 3-level image grid data 

structure. 

Two images 1^ and I2 include first level image grids G-| ^gn ^2 1st' second 
level image grids 2nd' ^2 2nd' ^^^"^^ '^^^9® Q^ds 3^^, G2 3rd- 
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The similarities Grid_Sim(G-|, G2) between grid levels included in two images 
are compared between the levels. The above-describe operation may be expressed 
as follows. 



Grid _Sim (G^.G^) ~w^x Sim _ of the Exact 
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where w1 through w9 represent weights with respect to the respective color 
similarity, and Sim_of_the_Exact represents a similarity between the same image grid 
levels with respect to two images I2, and Sim_of_theJnter represents a similarity 
10 between different image grid levels with respect to two images ^ , I2. 

Namely, the similarity Sim_ofJhe_Exact between the same image grid levels 
included in two different images 11 and 12 is obtained based on the match as shown in 
Figure 3. In addition, the similarity Sim„of_the_lnter between different image grid 
levels included in two different images 11 and 12 is obtained based on the match as 
15 shown in Figure 4. 

The above-described operation will be explained in more detail with reference 
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to Figures 5A and 5B. 

The similarities of two cells corresponding to the same levels of two different 
images are summed, and the similarities of two cells are summed to the thusly 
summed value by shifting in the horizontal and vertical directions by the aspect ratio. 
5 At this time, the number of the matches of two grids is computed by adding 1 to 

the absolute value of the difference of the aspect ratio of a certain level of two images. 

For example, as shown in Figure 5A, assuming that the number of the grids of 
the aspect ratio of the image l-| is MxN, and the number of the grids of the aspect ratio 
of the image I2 is OxP, the total number of matches between two grids is (|M- 
2io Ol)+1)x(|N-P|+1). 

; The similarity between two cells corresponding to the same grid levels 

t Max(M,N)=Max(0,P) is calculated by matching two grids based on different shift 

amount in accordance with the aspect ratio of two grids, 
f At this time, the similarity Sim_of_the_Exact based on the match between the 

fi5 same levels of two images 11 and 12 is obtained based on the following Equations 3-1 , 
f 3-2. 



Sim _of _the _Exact - Max {Sim _bet _two _levels given _cell _corres S(iJ)) 

Vi,0<i<|M-O| (3-1) 

Vj,0<i<|N-P| 

Sim_bet two levels given cell corres S(i, j) 

Min|N-P|-l Mm|M-0|-l 

^ ( ^ Sim _of _ corres _ two _ cells(x^ /, 7) (3-2) 

_ y=0 £=0 



Min{N,P)xMin{M,0) 



20 
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When matching the similarity (Sim_of_the_Exact) between the same levels, 



Mjn(\N-P\-l) Min(\M-0\-l) 

the above-described equation ^ ( ^Sim _of _corres _two _cells) represents 

y=0 x=0 



a sum of the matching with respect to the horizontal and vertical sides of two 
corresponding cells 

5 The similarity Sim_of_corrres_two_cells between two cells is obtained by 

adapting Equation 404 to Equation 4-1 based on the aspect ratios M:N, 0:P. 

Simicell''' (x + + j), cell''^ ( jc, y)\ if{Min{N, P) = P)r^ {Min{M, O) = O) ( 4- 1 ) 

Sim(ceir' (x + yXcell''^ (x,y + j)X if(Mm(N,P) = N)r^ (Mm(M, O) = O) (4-2) 

Simicell''' {x^y + i\ceU''^ {x + i,y)\ ifiMm(N,P) - P) (Mm(M, O) = M) (4-3 ) 

10 Sim{cell''\x,y\ceU''\x^Uy + J)X jf(Mm(N,P) = N)r^(Mm(M,0)=M) (4-4) 

Here, Equation 4-1 is applied when P is less than N and M is less than O and 
Equation 4-2 is applied when the length N of the grid is shorter than length P of 

grid G2 and the width O of the grid G2 is shorter than width M of the grid G1. In 



10 



addition, Equation 4-3 is applied when the vertical length P of the grid G2 is shorter 

than N of grid G1 and the horizontal length M of the grid G1 is shorter than O of G2, 
and Equation 4-4 is applied when N of G-] is shorter than P of G2 and M is shorter 

than O. 

5 At this time, the shift amount (ij) with respect to the length difference (|M-0|,|N- 

P|) between the length of the grid G^ and the grid G2 is added to the cell coordinate (x, 

y), and each of start point (ij,x,y) becomes 0. 

The similarity Sim_of_the_lnter between different grid levels(Max(M,N) 
^Max(0,P) is calculated by matching two different image grid levels. This operation is 

10 performed similarly as the search of the grid level similarity Sim__of_the__Exact. 

In addition, the number of the matches of the image grids between different 
image grid levels is obtained based on (|M-0|+1)x(|N-P|+1). 

The color region matching operation is performed for searching the region in 
which the representative color values are similar between the multilevel image grids. 

15 The search is performed based on a method for searching the color similarity from a 
translation position and a relative position between the grid level(Exact scale 
matching) of the same size, and a method for searching the color similarity from a 
translation position and the relative position between the grid levels( Inter-scale 
matching) of different sizes. 

20 Namely, the color region matching operation between the image grid 

levels{Exact scale matching) of the same size is performed based on a method for 
searching a color region of the same levels from a target image. The position is 
matched with the relative position based on the same image grid level of the target 
image, and then the similarity of the color region is computed, and the position is 

25 matched with a translation position at the same level of the target image for thereby 
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computing a similarity of the color region. 

The color region matching operation between the different image grid 
levels{lnter-scale matching) is performed based on a method for searching the 
different level color regions among the target images, and a similarity of the color 
region of the same level is computed among the different image grid levels of the 
target image. 

In the color region matching method of between different image grid levels, the 
similarity of the color region is computed by matching the position with the same 
position among the different image grid levels of the target image, and the similarity of 
the color region is computed by matching the position with the translation position at 
another level of the target image. 

As described above, in the present invention, one image grid data structure is 
divided into multilevel grid data structures. Therefore, it is possible to effectively 
response with respect to a subjective query by a user when searching a content- 
based image using the divided multilevel grid structures. In addition, an image search 
speed is fast and accurate under a certain condition. 

Although the preferred embodiment of the present invention have been 
disclosed for illustrative purposes, those skilled in the art will appreciate that various 
modifications, additions and substitutions are possible, without departing from the 
scope and spirit of the invention as recited in the accompanying claims. 
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What is claimed is: 

1 . A multilevel image data structure which is characterized in that an image 
frame including features of the image is expressed based on an image grid having at 

5 least two different hierarchical levels. 

2. The structure of claim 1, wherein said hierarchical level image grid 
includes cells and is hierarchically divided, and each cell is assigned with the 

^ representative color and reliability wherein the representative cloir represents a color 
fio feature of the region corresponding to the cell and the reliabilty of the represent color. 

t 3. The structure of claim 1 , wherein said hierarchical level of the image grid 

uniformly divides with the same number for width and height in the case that an 
i original image has the same width and height. 

fl5 

I 4. The structure of claim 1 , wherein in said hierarchical level of the image 

grid, in the case that an original image has different size for width and height, one side 
is uniformly divided, and the other side is divided based on the dividing unit of the one 
side. 

20 

5. An image search method using a multilevel image data structure, 
comprising the steps of: 

matching a spatial color feature of a reference image and target image, which 
are represented to different hierarchical image grid levels; and 
25 searching images based on a content-based query by a user. 
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6. The method of claim 5, wherein said color similarity between two images 
having different hierarchical grid levels is obtained by matching each cell included in 
two different image grids and based on a similarity between the representative color 
values having a spatial color feature. 

5 

7. The method of claim 5, wherein said color similarity between two images 
having different hierarchical grids is obtained by matching two image grids, performing 
a multi-cross in accordance with a spacious color feature between images and 
comparing a color similaries. 

fio 

; 8. The method of claim 5, wherein a color similarity between two images 

I having different hierarchical grids is obtained by matching each region representative 
color value for thereby searching the similar regions. 

:15 9. The method of claim 5, wherein a cell similarity between cells included in 

i the image grid having different hierarchical levels is obtained by multiplying the color 
similarity (Color^Sim) corresponding to a similarity of the region representative colors 
between two cells and the first weight, adding a value obtained by multiplying the 
similarity(l) representing a similarity of a reliability between two cells and a second 
20 weight to the color similarity (Color_Sim), and normalizing the similarity. 

10. The method of claim 5, wherein said color similarity betwen the two 
same level grids is obtained based on the total value summed by shifting in a 
horizontal and vertical direction based on the shifting amount by the difference of the 
25 widths and heights between grids when two grids are compared and the similaity is 
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calculated. 



11. The method of claim 5, wherein a color similarity between the two 
different grids is obtained based on a value summed shifting in a horizontal and 
vertical direction by the difference of the width and heights between the grids. 

12. The method of claim 5, wherein a cell similarity between image grids 
having a multilevel is used for searching the same position and different position 
between the same levels between the images in the case that the search is performed 
by matching the color region. 

13. The method of claim 5, wherein a color region matching operation 
between two image grids having a multilevel is directed to searching at the same 
position of different levels and at different position when searching the color similarity 
between different levels. 
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ABSTRACT OF THE DISCLOSURE 



The present invention relates to an image search method capable of 
expressing one color feature related to a spatial color feature of a still image based on 
5 a multilevel image grid and similarity-based searching images using the thusly 
expressed multilevel image grid. In the present invention, hierarchical grids of different 
levels are generated with respect to one feature for thereby obtaining a data structure 
in which each cell corresponding to the grid is expressed based on a reliability on a 
region representative color and the region representative color, so that it is possible to 
10 fast and accurately search images with respect to a user's content-based query based 
on a cell matching of the same level as two image grids and different levels or a color 
local matching of the grid match. 

15 
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FIG. 1 
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prior United States or PCT intemational appHcation in the manner provided by the first paragraph of 35 U.S.C. 112, 1 acknowledge the duty 
to disclose information which is material to patentabiHty as defined in 37 CFR 1.56 which became available between the fiHng date of the 
prior appHcation and the national or PCT intemational filing date of this appHcation. 

Prior U. S. Application 

or PCT Parent Mum ber Filing Hate (Month /Dav/Year^ Parent Patent N^imber (if applicab l e) 



I hereby declare that aH statements made herein of my own knowledge are tme and that all statements made on information and beHef are 
beHeved to be true; and fiirther that these statements were made with the knowledge that wiHful false statements ar:d the Hke so made are 
pumshabie by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such wiUfixl false statements 
may jeopardize the vaHdity of the appHcation or any patent issued thereon. 



I hereby appoint the foUowing attomey(s) and/or ageiit(s): Daniel YJ. Kim, Registration No. 36,186 and Mark L. Reshner, Registration No. 
34 596; Carl R. Wesolowski, Registration No. 40,372, John C. Eisenhart, Registration No. 38,128, Rene A. Vazquez, Registration No. 38,647; 
Michael J. ComeKson, Registration No. 40,395; and Stuart L Smitii, Registration No. 42,159; and Carol L. Druzbick, Registration No. 40,287, 
aUof 

FLESHNER & KIM 

P. O. Box 221200 

ChantiHy, Virginia 20153-1200 

with fbU power of substitution and revocation, to prosecute tiiis appHcation and to transact aU business in the Patent and Trademark Office 
coimected therewith, and all future correspondence should be addressed to them. 

Full name of sole or first inventor: Hyeon Jun KIM 



Inventor's si^ature: ^"^^^^ ^ Date: / > 

Residence: Siingnam, Korea ^ . ■ 



Citizenship: Republic of Korea 



Post Office Address: Hansin Life Apt. 109-302, Bundang-Dong, Bundang-Ku, Sungnam, 
Kyungki-Do, Korea 



Full name of joint inventor(s) : Sung Bae J UN 



Inventor's signature: :^J^&P] Date: ><^, '-V 



, ^ ! Residence: Seoul, Korea 



y1 Citizenship: Republic of Korea . 

" " Post Office Address: 804, Siheung 4Dong, Siheun g, Kumchun-Ku, Seoul, Korea 

1^ *>,.,;,>^>K>^************************=^^****>^****** 



Full name of joint invcntor(s) : Jin Soo LEE 



T . . • . ' Date:><^- ' 
Inventor s signature: — — 

Residence: Seoul, Korea' 



Citizenship: Republic of Korea 



Post Office Address: Samik Apt 101-804, 573, Machun 2Dong, Songpa-Ku, Seoul, Korea 



Full name of joiut inventor(s): ^ — . 



Inventor's signature: ^ _ T>^t^- 



Residence: 



Citizenship: 



Post Office Address: 



^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^ 

Full name of joint inventor (s): ^ — _ — 



Inventor's signature: 



Residence: 
Citizenship: 



Post Office Address: 



